Introduction
In this unit, we examine the backward pass of neural networks, which is central to the training process.
Today's Focus:
- Backward Pass (Scalar): Understanding gradient computation for individual weights
- Walkthrough Example: Step-by-step calculation with concrete numbers
- Matrix Notation: Efficient computation for entire layers
- Python Implementation: Translating theory into code
Together, these steps connect the mathematical backward pass to an implementation that can be used to train a neural network.
This lecture builds on the forward propagation process and introduces backward propagation, which enables the network to learn by computing gradients of the loss with respect to its parameters.
The lecture then develops the backward pass from individual gradients to matrix-based computations and a Python implementation.
Theory
Recap: Cost Function
As in logistic regression, we use a loss function to measure how far the network's output \(\hat{y}\) is from the true label \(y\).
Binary Cross-Entropy Loss (for a single example):
\[ L(y, \hat{y}) = -[y \cdot \ln(\hat{y}) + (1 - y) \cdot \ln(1 - \hat{y})] \]For m samples:
\[ L = -\frac{1}{m} \sum_{i=1}^{m} [y_i \cdot \ln(\hat{y}_i) + (1 - y_i) \cdot \ln(1 - \hat{y}_i)] \]Important: The network output is a function of the inputs and all weights and biases in the hidden and output layers:
\[
\hat{y} = \sigma \left(b_o + \sum_j w_{h_j,o} \sigma \left(b_{h_j} + \sum_i w_{i,h_j} x_i\right)\right)
\]
Key Insight: The cost function \(L\) depends on all the parameters (weights and biases) in the network. To train the network, we need to compute how much each parameter contributed to the error, which is what the backward pass does.
Example Dataset
Consider a small dataset of six observations with two features, Fat Score and Salt Score, and a binary outcome, Acceptance (like/dislike).
| Obs. | Fat score | Salt score | Acceptance |
|---|---|---|---|
| 1 | 0.6 | 0.4 | like |
| 2 | 0.1 | 0.1 | dislike |
| 3 | 0.2 | 0.4 | dislike |
| 4 | 0.2 | 0.5 | dislike |
| 5 | 0.4 | 0.5 | like |
| 6 | 0.3 | 0.8 | like |
Network Architecture:
- Input layer: 2 features (Fat score, Salt score) \(\in \mathbb{R}^2\)
- Hidden layer: 3 units \(\in \mathbb{R}^3\)
- Output layer: 1 unit (probability of "like") \(\in \mathbb{R}^1\)
Note: If we remove the hidden layer (directly connect inputs to output with a sigmoid), we would essentially have logistic regression on Fat & Salt. The hidden layer lets us build a more expressive function.
Initialization
For the numerical example, we use the first observation: \(x_1 = 0.6, x_2 = 0.4\).
Hidden Layer (3 neurons \(h_1, h_2, h_3\)):
- Neuron \(h_1\): \[ z_{h1} = b_{h1} + w_{1,h1}x_1 + w_{2,h1}x_2 \] \[ a_{h1} = \sigma(z_{h1}) \]
- Neuron \(h_2\): \[ z_{h2} = b_{h2} + w_{1,h2}x_1 + w_{2,h2}x_2 \] \[ a_{h2} = \sigma(z_{h2}) \]
- Neuron \(h_3\): \[ z_{h3} = b_{h3} + w_{1,h3}x_1 + w_{2,h3}x_2 \] \[ a_{h3} = \sigma(z_{h3}) \]
Initial weights and biases:
| Parameter | Value |
|---|---|
| Weights to \(h_1\): | \(w_{1,h1} = 0.5\), \(w_{2,h1} = -0.4\) |
| Bias \(b_{h1}\): | 0.1 |
| Weights to \(h_2\): | \(w_{1,h2} = -0.3\), \(w_{2,h2} = 0.8\) |
| Bias \(b_{h2}\): | -0.2 |
| Weights to \(h_3\): | \(w_{1,h3} = 0.2\), \(w_{2,h3} = 0.1\) |
| Bias \(b_{h3}\): | 0.05 |
| Weights from hidden to output: | \(w_{h1,o} = 0.7\), \(w_{h2,o} = -0.6\), \(w_{h3,o} = 0.3\) |
| Bias \(b_o\): | 0.2 |
Interactive Examples
Forward Pass
The forward pass computes the network's output in a sequence of steps:
\[
x_1, x_2 \to [z_j = \sum w \cdot x + b] \to [a_j = \sigma(z_j)] \to [z_o = \sum w \cdot a + b] \to [\hat{y} = \sigma(z_o)] \to L
\]
Hidden Layer Computations:
\[
\begin{array}{l}
z_{h1} = b_{h1} + w_{1,h1}x_1 + w_{2,h1}x_2 = 0.1 + (0.5)(0.6) + (-0.4)(0.4) = 0.1 + 0.30 - 0.16 = 0.24 \\
a_{h1} = \sigma(0.24) \approx 0.5597
\end{array}
\]
\[
\begin{array}{l}
z_{h2} = b_{h2} + w_{1,h2}x_1 + w_{2,h2}x_2 = -0.2 + (-0.3)(0.6) + (0.8)(0.4) = -0.2 - 0.18 + 0.32 = -0.06 \\
a_{h2} = \sigma(-0.06) \approx 0.4850
\end{array}
\]
\[
\begin{array}{l}
z_{h3} = b_{h3} + w_{1,h3}x_1 + w_{2,h3}x_2 = 0.05 + (0.2)(0.6) + (0.1)(0.4) = 0.05 + 0.12 + 0.04 = 0.21 \\
a_{h3} = \sigma(0.21) \approx 0.5523
\end{array}
\]
Output Layer Computation:
\[
\begin{array}{l}
z_o = b_o + w_{h1,o}a_{h1} + w_{h2,o}a_{h2} + w_{h3,o}a_{h3} \\
= 0.2 + (0.7)(0.5597) + (-0.6)(0.4850) + (0.3)(0.5523) \\
= 0.2 + 0.3918 - 0.2910 + 0.1657 = 0.4665
\end{array}
\]
\[
\hat{y} = \sigma(0.4665) \approx 0.6147
\]
Result: The network's output \(\hat{y} \approx 0.6147\), which could be interpreted as the probability of "like".
Backward Pass: Update Rules
During training, gradient descent updates the parameters using the gradients computed in the backward pass:
- Output Bias: \(b_o \gets b_o - \alpha \frac{\partial L}{\partial b_o}\)
- Weights from hidden to output: \(w_{h_j,o} \gets w_{h_j,o} - \alpha \frac{\partial L}{\partial w_{h_j,o}}\) for each j
- Hidden biases: \(b_{h_j} \gets b_{h_j} - \alpha \frac{\partial L}{\partial b_{h_j}}\) for each j
- Weights from Input to Hidden: \(w_{i,h_j} \gets w_{i,h_j} - \alpha \frac{\partial L}{\partial w_{i,h_j}}\) for each i, j
Important Note: Use the original parameter values from the forward pass when computing all gradients. The gradients should be based on the same network state before any parameter updates occur.
Theory: Backward Pass Equations
Step 1: Output Layer Gradient
Gradient for output bias:
\[ \frac{\partial L}{\partial b_o} = \hat{y} - y \]Gradient for weights from hidden to output:
\[ \frac{\partial L}{\partial w_{h_j,o}} = (\hat{y} - y) \cdot a_j \quad \text{for each } j \]Where:
- \(a_j\): activation of hidden neuron \(h_j\)
- \(w_{h_j,o}\): weight from hidden neuron j to output
Step 2: Hidden Layer Gradients
2a: Error at hidden layer activations:
\[ \frac{\partial L}{\partial a_j} = (\hat{y} - y) \cdot w_{h_j,o} \]2b: Error at hidden pre-activation (chain rule):
\[ \frac{\partial L}{\partial z_j} = \frac{\partial L}{\partial a_j} \cdot \sigma'(z_j) = \frac{\partial L}{\partial a_j} \cdot a_j \cdot (1 - a_j) \]2c: Gradient for hidden bias:
\[ \frac{\partial L}{\partial b_{h_j}} = \frac{\partial L}{\partial z_j} \]2d: Gradient for hidden weights:
\[ \frac{\partial L}{\partial w_{i,h_j}} = \frac{\partial L}{\partial z_j} \cdot x_i \quad \text{for each input } i \]Numerical Solutions
Backward Pass Working Example
For this example, \(\hat{y} \approx 0.6147\), the true label is \(y = 1\), and the learning rate is \(\eta = 0.1\).
Step 1: Compute error term at output
\[
\delta_o = \hat{y} - y = 0.6147 - 1 = -0.3853
\]
(Note: this arises from \(\frac{\partial L}{\partial z_o}\) given cross-entropy + sigmoid.)
Step 2: Update output weight \(w_{h1,o} = 0.7\) (initial)
\[
\frac{\partial L}{\partial w_{h1,o}} = \delta_o \cdot a_{h1} = (-0.3853) \cdot (0.5597) \approx -0.2156
\]
Weight update:
\[ w_{h1,o}^{\text{new}} = w_{h1,o}^{\text{old}} - \eta \times \frac{\partial L}{\partial w_{h1,o}} = 0.7 - 0.1 \times (-0.2156) = 0.7 + 0.02156 = 0.72156 \]Update bias \(b_o\):
\[ \frac{\partial L}{\partial b_o} = \delta_o \cdot 1 = -0.3853 \] \[ b_o^{\text{new}} = 0.2 - 0.1 \times (-0.3853) = 0.2 + 0.03853 = 0.23853 \]Step 3: Back-propagate to hidden layer
We now compute the terms for hidden neuron \(h_2\):
The forward pass gave \(a_{h2} \approx 0.4850\) and \(z_{h2} \approx -0.06\).
Compute hidden neuron error term:
\[ \delta_{h2} = \delta_o \cdot w_{h2,o} \cdot a_{h2} (1 - a_{h2}) = (-0.3853) \cdot (-0.6) \cdot (0.4850 \times 0.5150) \] \[ = 0.23118 \cdot (0.2498) \approx 0.0577 \]Gradient for \(w_{1,h2}\) (input \(x_1 = 0.6\)):
\[ \frac{\partial L}{\partial w_{1,h2}} = \delta_{h2} \cdot x_1 = 0.0577 \cdot 0.6 \approx 0.03462 \]Update \(w_{1,h2}\):
\[ w_{1,h2}^{\text{new}} = -0.3 - 0.1 \times (0.03462) = -0.3 - 0.003462 = -0.303462 \]Update bias of \(h_2\):
\[ \frac{\partial L}{\partial b_{h2}} = \delta_{h2} \cdot 1 = 0.0577 \] \[ b_{h2}^{\text{new}} = -0.2 - 0.1 \times 0.0577 = -0.2 - 0.00577 = -0.20577 \]Theory: Forward and Backward Pass in Matrix Notation
Forward Pass (Matrix Notation)
Hidden Layer:
\[ z^{[1]} = (W^{[1]})^T x + b^{[1]} \] \[ a^{[1]} = \sigma(z^{[1]}) \]Output Layer:
\[ z^{[2]} = (W^{[2]})^T a^{[1]} + b^{[2]} \] \[ \hat{y} = \sigma(z^{[2]}) \]Dimensions:
\[
\begin{array}{l}
W^{[1]} \in \mathbb{R}^{2 \times 3}, \quad x \in \mathbb{R}^{2 \times 1}, \quad b^{[1]} \in \mathbb{R}^{3 \times 1} \\
W^{[2]} \in \mathbb{R}^{3 \times 1}, \quad b^{[2]} \in \mathbb{R}^{1 \times 1}
\end{array}
\]
Backward Pass (Matrix Notation)
Output Layer Gradients:
\[ \frac{\partial L}{\partial b^{[2]}} = \hat{y} - y \] \[ \frac{\partial L}{\partial W^{[2]}} = (\hat{y} - y) (a^{[1]})^T \]Hidden Layer Gradients:
\[ \frac{\partial L}{\partial b^{[1]}} = W^{[2]} (\hat{y} - y) \odot a^{[1]} \odot (1 - a^{[1]}) \] \[ \frac{\partial L}{\partial W^{[1]}} = \left[ W^{[2]} (\hat{y} - y) \odot a^{[1]} \odot (1 - a^{[1]}) \right] x^T \]Here, \(\odot\) denotes element-wise multiplication.
Key Insight: Matrix notation allows us to compute all gradients for a layer in a single operation, rather than computing each weight's gradient individually. This is much more efficient, especially for large networks.
Python Implementation
Helper Functions
def sigmoid(z):
return 1.0 / (1 + np.exp(-z))
def sigmoid_derivative(z):
return sigmoid(z) * (1.0 - sigmoid(z))
def predict(x, model):
W1 = model['W1']
b1 = model['b1']
W2 = model['W2']
b2 = model['b2']
Z1 = np.matmul(W1, x) + b1
A1 = sigmoid(Z1)
Z2 = np.matmul(W2, A1) + b2
A2 = sigmoid(Z2)
return A2
Training Function
def train(X, y, m_hidden, learning_rate, n_iter):
n_input, m = X.shape
# Initialize weights
W1 = np.random.randn(m_hidden, n_input)
b1 = np.zeros((m_hidden, 1))
W2 = np.random.randn(1, m_hidden)
b2 = np.zeros((1, 1))
for i in range(1, n_iter + 1):
# ==== Forward pass ====
Z1 = np.matmul(W1, X) + b1
A1 = sigmoid(Z1)
Z2 = np.matmul(W2, A1) + b2
A2 = sigmoid(Z2) # y_hat
# ==== Backward pass ====
dZ2 = A2 - y
dW2 = np.matmul(dZ2, A1.T) / m
db2 = np.sum(dZ2, axis=1, keepdims=True) / m
dA1 = np.matmul(W2.T, dZ2)
dZ1 = dA1 * sigmoid_derivative(Z1)
dW1 = np.matmul(dZ1, X.T) / m
db1 = np.sum(dZ1, axis=1, keepdims=True) / m
# ==== Parameter update ====
W1 -= learning_rate * dW1
b1 -= learning_rate * db1
W2 -= learning_rate * dW2
b2 -= learning_rate * db2
# Optional training log
if i % 100 == 0:
cost = np.mean((y - A2)**2)
print(f"Iteration {i}, training loss: {cost}")
return {'W1': W1, 'b1': b1, 'W2': W2, 'b2': b2}
Key Implementation Notes:
- Vectorized operations: The implementation uses matrix operations (np.matmul) for efficiency
- Sigmoid derivative: Uses the property that \(\sigma'(z) = \sigma(z) \cdot (1 - \sigma(z))\)
- Gradient computation: Follows the chain rule to compute gradients for each layer
- Weight updates: Uses gradient descent with the specified learning rate
- Training log: Optionally prints the loss every 100 iterations
These implementation choices translate the matrix-based backward pass into a compact training procedure.
Try It Yourself
Problem 1: Forward Pass Calculation
Given the following network parameters and input:
- Input: \(x_1 = 0.5, x_2 = 0.3\)
- Hidden neuron \(h_1\): \(w_{1,h1} = 0.4, w_{2,h1} = -0.2, b_{h1} = 0.1\)
- Output neuron: \(w_{h1,o} = 0.6, b_o = 0.2\)
Tasks:
- Calculate \(z_{h1}\) and \(a_{h1}\)
- Calculate \(z_o\) and \(\hat{y}\)
Solution:
- Hidden neuron calculation: \[ z_{h1} = 0.1 + (0.4)(0.5) + (-0.2)(0.3) = 0.1 + 0.2 - 0.06 = 0.24 \] \[ a_{h1} = \sigma(0.24) \approx 0.5597 \]
- Output neuron calculation: \[ z_o = 0.2 + (0.6)(0.5597) = 0.2 + 0.3358 = 0.5358 \] \[ \hat{y} = \sigma(0.5358) \approx 0.6306 \]
Problem 2: Backward Pass Calculation
Using the network from Problem 1, assume:
- True label \(y = 1\)
- Learning rate \(\eta = 0.1\)
- Predicted output \(\hat{y} \approx 0.6306\)
Tasks:
- Calculate the error term \(\delta_o\)
- Calculate the gradient \(\frac{\partial L}{\partial w_{h1,o}}\)
- Update the weight \(w_{h1,o}\)
- Update the bias \(b_o\)
Solution:
- Error term: \[ \delta_o = \hat{y} - y = 0.6306 - 1 = -0.3694 \]
- Gradient for \(w_{h1,o}\): \[ \frac{\partial L}{\partial w_{h1,o}} = \delta_o \cdot a_{h1} = (-0.3694) \cdot (0.5597) \approx -0.2075 \]
- Update \(w_{h1,o}\): \[ w_{h1,o}^{\text{new}} = 0.6 - 0.1 \times (-0.2075) = 0.6 + 0.02075 = 0.62075 \]
- Update \(b_o\): \[ \frac{\partial L}{\partial b_o} = \delta_o = -0.3694 \] \[ b_o^{\text{new}} = 0.2 - 0.1 \times (-0.3694) = 0.2 + 0.03694 = 0.23694 \]
Problem 3: Matrix Notation Forward Pass
Given:
- Input vector \(x = \begin{bmatrix} 0.5 \\ 0.3 \end{bmatrix}\)
- Weight matrix \(W^{[1]} = \begin{bmatrix} 0.4 & -0.2 \\ -0.1 & 0.3 \\ 0.2 & 0.1 \end{bmatrix}\)
- Bias vector \(b^{[1]} = \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix}\)
Task: Calculate \(z^{[1]}\) and \(a^{[1]}\) using matrix notation.
Solution:
Calculate \(z^{[1]}\):
\[
z^{[1]} = (W^{[1]})^T x + b^{[1]} = \begin{bmatrix} 0.4 & -0.1 & 0.2 \\ -0.2 & 0.3 & 0.1 \end{bmatrix} \begin{bmatrix} 0.5 \\ 0.3 \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix}
\]
\[
= \begin{bmatrix} (0.4)(0.5) + (-0.1)(0.3) + (0.2)(?) \\ (-0.2)(0.5) + (0.3)(0.3) + (0.1)(?) \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix}
\]
Note: There seems to be a dimension mismatch. \(W^{[1]}\) should be \(2 \times 3\) (input features × hidden units), so \((W^{[1]})^T\) is \(3 \times 2\), and \(x\) is \(2 \times 1\), giving \(z^{[1]}\) as \(3 \times 1\).
\[ z^{[1]} = \begin{bmatrix} 0.4 & -0.2 \\ -0.1 & 0.3 \\ 0.2 & 0.1 \end{bmatrix}^T \begin{bmatrix} 0.5 \\ 0.3 \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix} \] \[ = \begin{bmatrix} 0.4 & -0.1 & 0.2 \\ -0.2 & 0.3 & 0.1 \end{bmatrix} \begin{bmatrix} 0.5 \\ 0.3 \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix} \] \[ = \begin{bmatrix} (0.4)(0.5) + (-0.1)(0.3) \\ (-0.2)(0.5) + (0.3)(0.3) \\ (0.2)(0.5) + (0.1)(0.3) \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix} \] \[ = \begin{bmatrix} 0.2 - 0.03 \\ -0.1 + 0.09 \\ 0.1 + 0.03 \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix} = \begin{bmatrix} 0.28 \\ -0.02 \\ 0.18 \end{bmatrix} \]Calculate \(a^{[1]}\): Apply sigmoid to each element of \(z^{[1]}\)
\[
a^{[1]} = \sigma(z^{[1]}) = \begin{bmatrix} \sigma(0.28) \\ \sigma(-0.02) \\ \sigma(0.18) \end{bmatrix} \approx \begin{bmatrix} 0.5698 \\ 0.4950 \\ 0.5445 \end{bmatrix}
\]
Problem 4: Backward Pass Matrix Notation
Given:
- Output \(\hat{y} = 0.7\), true label \(y = 1\)
- Hidden layer activations \(a^{[1]} = \begin{bmatrix} 0.6 \\ 0.4 \\ 0.5 \end{bmatrix}\)
- Output weights \(W^{[2]} = \begin{bmatrix} 0.5 & -0.3 & 0.2 \end{bmatrix}\)
Tasks:
- Calculate \(\frac{\partial L}{\partial b^{[2]}}\)
- Calculate \(\frac{\partial L}{\partial W^{[2]}}\)
Solution:
- Output bias gradient: \[ \frac{\partial L}{\partial b^{[2]}} = \hat{y} - y = 0.7 - 1 = -0.3 \]
- Output weight gradient: \[ \frac{\partial L}{\partial W^{[2]}} = (\hat{y} - y) (a^{[1]})^T = (-0.3) \begin{bmatrix} 0.6 & 0.4 & 0.5 \end{bmatrix} = \begin{bmatrix} -0.18 & -0.12 & -0.15 \end{bmatrix} \]
Problem 5: Chain Rule Application
Consider a simple network with:
- Input \(x\)
- Hidden layer: \(z_h = w_1 x + b_1\), \(a_h = \sigma(z_h)\)
- Output layer: \(z_o = w_2 a_h + b_2\), \(\hat{y} = \sigma(z_o)\)
- Loss: \(L = (\hat{y} - y)^2\)
Task: Derive the expression for \(\frac{\partial L}{\partial w_1}\) using the chain rule.
Solution:
Using the chain rule:
\[
\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z_o} \cdot \frac{\partial z_o}{\partial a_h} \cdot \frac{\partial a_h}{\partial z_h} \cdot \frac{\partial z_h}{\partial w_1}
\]
Compute each term:
- \(\frac{\partial L}{\partial \hat{y}} = 2(\hat{y} - y)\)
- \(\frac{\partial \hat{y}}{\partial z_o} = \sigma'(z_o) = \hat{y}(1 - \hat{y})\)
- \(\frac{\partial z_o}{\partial a_h} = w_2\)
- \(\frac{\partial a_h}{\partial z_h} = \sigma'(z_h) = a_h(1 - a_h)\)
- \(\frac{\partial z_h}{\partial w_1} = x\)
Combine:
\[
\frac{\partial L}{\partial w_1} = 2(\hat{y} - y) \cdot \hat{y}(1 - \hat{y}) \cdot w_2 \cdot a_h(1 - a_h) \cdot x
\]
Problem 6: Dropout Implementation
You have a hidden layer with 100 neurons and want to apply dropout with a rate of 0.25.
Tasks:
- How many neurons will be kept (on average) in each training iteration?
- What is the probability that a specific neuron is dropped?
- During inference (testing), if a neuron has an activation of 0.8, what will be its scaled output?
Solution:
- Neurons kept: 100 × (1 - 0.25) = 75 neurons (on average)
- Probability of dropping: 0.25 (dropout rate)
- Scaled output during inference: At test time, dropout is turned off, but outputs are scaled by the dropout rate to maintain expected values. So 0.8 × (1 - 0.25) = 0.8 × 0.75 = 0.6
Problem 7: Learning Rate Scheduling
You are training a neural network with an initial learning rate of \(\eta_0 = 0.1\).
Tasks:
- Using step decay with a factor of 0.5 every 100 iterations, what is the learning rate at iteration 250?
- Using exponential decay with \(k = 0.01\), what is the learning rate at iteration 100?
- Using 1/t decay with \(k = 0.1\), what is the learning rate at iteration 50?
Solution:
- Step decay: At iteration 250, we've passed 2 decay points (100 and 200). Learning rate = 0.1 × (0.5)^2 = 0.1 × 0.25 = 0.025
- Exponential decay: \(\eta = \eta_0 \cdot e^{-kt} = 0.1 \cdot e^{-0.01 \times 100} = 0.1 \cdot e^{-1} \approx 0.1 \times 0.3679 = \) 0.03679
- 1/t decay: \(\eta = \eta_0 / (1 + kt) = 0.1 / (1 + 0.1 \times 50) = 0.1 / (1 + 5) = 0.1 / 6 \approx \) 0.01667
Interactive Quiz
Use the following questions to check your understanding of the backward pass in neural networks.
Score: 0 / 5
Key Takeaways
Backward Pass Fundamentals:
- Purpose: Computes the gradients of the loss function with respect to all parameters (weights and biases)
- Mechanism: Uses the chain rule to propagate the error backward through the network
- Order: Compute output-layer gradients first, followed by hidden-layer gradients
- Synchronous updates: Compute all gradients using the same network state (the original parameter values) before applying any updates
Output Layer Gradients:
- Bias gradient: \(\frac{\partial L}{\partial b_o} = \hat{y} - y\)
- Weight gradient: \(\frac{\partial L}{\partial w_{h_j,o}} = (\hat{y} - y) \cdot a_j\) for each hidden unit j
- Error term: \(\delta_o = \hat{y} - y\) (for cross-entropy + sigmoid)
Hidden Layer Gradients:
- Activation error: \(\frac{\partial L}{\partial a_j} = (\hat{y} - y) \cdot w_{h_j,o}\)
- Pre-activation error: \(\frac{\partial L}{\partial z_j} = \frac{\partial L}{\partial a_j} \cdot \sigma'(z_j) = \frac{\partial L}{\partial a_j} \cdot a_j \cdot (1 - a_j)\)
- Bias gradient: \(\frac{\partial L}{\partial b_{h_j}} = \frac{\partial L}{\partial z_j}\)
- Weight gradient: \(\frac{\partial L}{\partial w_{i,h_j}} = \frac{\partial L}{\partial z_j} \cdot x_i\)
Matrix Notation:
- Efficiency: Allows all gradients for a layer to be computed in a single operation
- Forward pass: \(z^{[l]} = (W^{[l]})^T a^{[l-1]} + b^{[l]}\), \(a^{[l]} = \sigma(z^{[l]})\)
- Backward pass: Uses element-wise operations (\(\odot\)) to compute gradients efficiently
Python Implementation:
- Vectorized: Uses matrix operations for efficient computation
- Sigmoid derivative: \(\sigma'(z) = \sigma(z) \cdot (1 - \sigma(z))\)
- Chain rule: Uses the chain rule to compute gradients layer by layer
- Weight updates: Updates weights using \(w \gets w - \eta \cdot \frac{\partial L}{\partial w}\)
Common Pitfalls
Backward Pass:
- Using updated weights: Computing gradients with updated weights instead of the original weights can produce incorrect gradient estimates
- Incorrect chain rule application: Omitting terms in the chain rule can lead to incorrect gradients
- Order of operations: Computing hidden-layer gradients before output-layer gradients breaks the dependency chain
- Numerical instability: Very small or very large gradients can cause numerical issues
- Vanishing gradients: For deep networks, gradients can become extremely small, causing early layers to learn very slowly
Matrix Notation:
- Dimension mismatch: Incorrect matrix dimensions can cause errors in gradient computation
- Transpose errors: Forgetting a required transpose can lead to incorrect matrix dimensions
- Element-wise vs matrix multiplication: Confusing \(\odot\) (element-wise multiplication) with matrix multiplication
- Broadcasting issues: In NumPy, broadcasting can produce unexpected results if dimensions are not managed carefully
Implementation:
- Learning rate too large: Can cause weights to oscillate or the training process to diverge
- Learning rate too small: Can lead to very slow convergence
- Not initializing weights: Starting all weights at zero can prevent the network from learning effectively
- Numerical precision: Limited numerical precision can cause issues in some cases
- Debugging: Backward-pass implementations can be difficult to debug, so unit tests are useful for checking the gradients
Conceptual:
- Confusing forward and backward: The forward pass computes outputs, whereas the backward pass computes gradients
- Local vs global minima: Gradient descent finds local minima, not necessarily global minima
- Convexity: Neural network loss functions are typically non-convex, having many local minima